Concerning the difficulty of its similarity to the expression and the effects of noise in medical image retrieval, a diffusion-based approach on a tensor product graph was proposed to improve the texton-based pairwise similarity metric by context information of other database objects. Firstly, medical image features were described and extracted by texton-based statistical method, and then the pairwise similarities were obtained with weights determined by the similarities between textons. A global similarity metric was achieved by utilizing the tensor product graph to propagate the similarity information along the intrinsic structure of the data manifold. Experimental results of ImageCLEFmed 2009 database show that, the proposed algorithm improves the performance by an average class accuracy of 32% and 19% compared with the Gabor-based retrieval algorithm and the Scale-Invariant Feature Transform (SIFT)-based retrieval algorithm respectively, which can be applied to medical image retrieval.